During the past twenty years, the data storage market has been characterized by a rush towards collaborative computing. With its focus on centralized storage, this collaborative computing trend has created an enormous market for various types of storage and storage management applications.
Beginning with the first local area network (LAN), the basic need to share data and files between computers on a LAN led to direct attached storage (DAS). As networks increased in complexity and scalability, applications evolved to make use of the additional capacity. Applications were able to support dozens and even hundreds of users. As networks scaled, applications became separated from their physical storage and data by distance. As a result, deployment of network attached storage (NAS) became commonplace and for the first time allowed files to be shared by simultaneous users on the network. More recently, the demand for ever-increasing storage capacity across heterogeneous networks has led to the advent of storage area networks (SANs).
SANs introduce a layer of storage management software that plays a vital role in administering distributed IT assets, maintaining high availability, and minimizing downtime. The appeal of the SAN architecture is that effective management of resources from the data center across the network reduces cost and increases both productivity and revenue. SANs and similar data storage solutions require implementation of data storage management tasks or functions such as storage space allocation, storage virtualization, cache management, fault tolerance and recovery, management of RAID arrays, and numerous other functions.
As LANs, wide area networks (WANs), and NAS evolved into the first SANs, network managers were determined to improve the character of storage data management. The first attempt at creating a SAN was to place specialized management appliances between the applications and the storage devices on the network. This method was initially appealing because it afforded immediate control over the data stream. The practice of placing storage management appliances in the data path became known as “in-band architecture”. However, as more SANs were deployed it became apparent that the costs and complexity of this architectural scheme were enormous. These factors were initially responsible for minimal use of SANs in all but the biggest enterprises.
Traditionally SANs required a separate dedicated infrastructure to interconnect applications and storage systems. The primary means for these interconnections were Fibre Channel (FC) networks that provided the transport layer for providing storage commands. Storage devices generally use Small Computer System Interface (SCSI) protocol as the standard protocol for processing storage commands. SCSI enables block data transfer between the applications and peripheral devices. SCSI connects initiators, which issue commands, to targets that respond to commands. Typically, the initiators are application clients, and targets are storage subsystems and disk drives. The SCSI architecture is described in SCSI Architectural Model-2 (SAM-2) being propagated by the International Committee for Information Technology Standards (INCITS), incorporated by reference herein in its entirety.
The SCSI protocol defines a logical unit within the target to execute a command. A target typically has multiple logical units distinguished by their logical unit numbers (LUNs). SCSI initiators use a structure known as a command descriptor block (CDB) to send commands, such as read a specific number of blocks. The SCSI protocol refers to a linked set of operations as a task. Examples of SCSI tasks include a read command, a write command, and an inquiry command, just to list a few. Storage management devices have been designed to process SCSI tasks and perform storage operations, such as RAID or storage virtualization and then deliver commands to physical devices. A Front End Transport (FET) receives SCSI commands using a transport specific protocol. A Back End Transport (BET) issues SCSI commands using the same or different transport specific protocol. A processing block processes tasks presented via one or more FETs and issues tasks to one or more BETs. There is no relationship as to the number of FETs, processing blocks and BETs. For example, more than one FET, each supporting a different transport protocol, can supply tasks to one or more processing blocks, each supporting a different application, and one or more BET can receive tasks from a processing block.
The SCSI command structure is well understood and supported by drives and operating systems. As a result, the SCSI command set has become the dominant protocol for disks, tape storage, and storage device management. The primary SCSI commands are described in SCSI Primary Commands-2 (SPC-2) being propagated by INCITS, incorporated by reference herein in its entirety. The device-type specific command sets are also propagated by INCITS.
The industry responded to the cost and architectural complexity associated with fibre channel by attempting to manage storage allocation and virtualization in software running on a separate, common, network topology referred to as Ethernet. Traditionally, this approach required an application to be running on each host and on each storage device. This practice is known as “out-of-band architecture”. Unfortunately, this approach offers only a slight improvement because it reduces overall data throughput and introduces unacceptable network latencies.
There have been a multitude of challenges associated with the various implementations of SANs. Most notably, the SANs presently available suffer from high implementation cost, poor performance and high added latency, lower mean time between failures, and high power dissipation, primarily due to implementation with multiple and dispersed circuit components and boards.
A typical SAN appliance sits in the storage data path and introduces undesirable latency. With the increased usage of IP-based storage solutions, a storage appliance has to process storage-over-IP commands and related messages. As the network link speed increases, the overhead associated with processing of storage and task management functions becomes prohibitively slow.
Accordingly, there is a need for improved performance of systems and circuits that effect data storage management functions. There is a need for an improved architecture that allows data flow in data storage applications to be processed at or near the wire speeds of the data interfaces of data storage devices or appliances.
A desirable solution would allow distributed processing of storage commands such that read and write functions can be performed extremely quickly while other time consuming processes can be off loaded to other processors, if available. However, developing a distributed processing system for each possible hardware architecture or transport protocol would be inefficient and costly. Needed is an a interface for distributed processing that provides the ability to interconnect SCSI processing modules independent of the underlying transport or hardware architecture.